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LSQ (LEAST SQUARES SUBROUTINE) 


DECUS Program Library Write-up 


DECUS No. 8-134 


This subroutine calculates the slope and intercept for 
= mxj + b 

by the method of least squares. The rms deviation of y, the correlation 
coefficient and a estimate of the percent error in the slope are also 
calculated. 

Specifications : The program requires 1. 5 pages of memory and the 
following page 0 locations: 

10-13 temporary storage 
100-113;156-160 data return from subroutine 
117-123 data for subroutine 
114-116;153-155;161-177 temporary storage 
The page zero locations may be changed as desired by changing the 
list which preceeds the program on the symbolic tape. The subroutine 
calls the subroutine FLOAT, and uses the floating point interpreter, 
version C. 

Description : Before calling the subroutine, the following locations 

are set up: 

117 N = the number of data points 

120 LX = starting location for X array. 

121 LY = starting location for Y array 

122 LCALC = starting location of YCALC array 
into which the calculated values of y are 

to be placed 

123 LDIF = starting location of DIF array into 
which the differences between Y and YCALC 
are to be placed. 
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The subroutine is called by an effective JMS I LSQ. On return to 
the calling program, the following will be available: 

100 S = slope 

103 B = intercept 

106 R = correlation coefficient 

111 ERROR = rms deviation in y 

156 TEM2 = % error in slope 

Method of Calculation : 

The slope and intercept are obtained using the usual expressions: 

SLOPE = (SUMX*SUMY - G*SUMXY)/(SUMX* * 2 - G*SUMXX) 

B = (SUMX*SUMXY - SUMY*SUMXX)/(SUMX* * 2 - G*SUMXX) 

where G is the number of points in floating point form. The correlation 
coefficient is calculated from: 



R = (SU MX * SU MY - G*SUMXY)/SQRT((SUMY**2-G*SUMYY)*(SUMX**2-G*SUMXX)) 
and the % error in the slope is estimated from: 

%ERROR = G*SQRT(SUM(((X i -X)*DIF i )**2))/(SUMX*SUMY - G*SUMXY) 

The quantity SUM(((Xj-X)* DIF^)* * 2) is collected as SUMXY on pg. 2 of 

the subroutine. SUMX is set equal to X, and TEM3 contains the numerator ^ 

from the slope calculation. The above method of estimating the error in 

slope makes the error independent of translation of the x axis and was 

su gg es t e d by E. C. Lupton, Jr. 
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/LEAST SQUARES SUBROUTINE 

IX = 10 

I Y= 1 1 

ICALC=12 

IDIF=13 

S= 100 

B= 103 

R= 106 

ERROR=111 

G= 1 14 

N= 1 17 

LX=120 

LY=121 

LCAL C= 122 

LDIF=123 

SUMYY =153 

TEM2=156 

T EM 3= 16 1 

SUMX=164 

SUMY=167 

SUMXY=172 

SUMXX=175 

CALL= 4400 

RETURN=5400 

FLOAT=4355 

FSQRT=2 

FLPT= 4407 


3400 

0000 

*3400 

LSQ, 

0 

3401 

7200 


CLA 

3402 

1117 


TAD N 

3403 

7041 


CIA 

3404 

3366 


DCA NEGN 

3405 

1 120 


TAD LX 

3406 

3010 


DCA IX 

3407 

1 121 


TAD LY 

3410 

301 1 


DCA IY 

3411 

4407 


FLPT 

3412 

5361 


FGET ZERO 

3413 

6164 


FPUT SUMX 

3414 

6167 


FPUT SUMY 

3415 

6175 


FPUT SUMXX 

3416 

6172 


FPUT SUMXY 

3417 

6153 


FPUT SUMYY 

3420 

0000 


FEXT 

3421 

4407 

BEGIN, 

FLPT 

3422 

5167 


FGET SUMY 

3423 

1411 


FADD I IY 

3424 

6167 


FPUT SUMY 

3425 

5410 


FGET I IX 

3426 

1 164 


FADD SUMX 

3427 

6164 


FPUT SUMX 

3430 

5410 


FGET I IX 

3431 

3410 


FMPY I IX 

3432 

1 175 


FADD SUMXX 


/INDEX 

/SET UP REGISTERS 


/STORE SUM Y 
/GET X 

/STORE SUM X 
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3433 

6175 

FPUT SUMXX 

/STORE SUM X*X 

3434 

5410 

FGET I IX 


3435 

341 1 

FMPY I IY 


3436 

1 172 

F ADD SUMXY 


3437 

6172 

FPUT SUMXY 

/STORE SUM X*Y 

3440 

541 1 

FGET I IY 


3441 

341 1 

FMPY I IY 


3442 

1 1 53 

FADD SUMYY 


3443 

6153 

FPUT SUMYY 

/STORE SUM Y*Y 

3444 

0000 

FEXT 

/FINISHED SUMMATIONS 

3445 

201 1 

ISZ IY 


3446 

201 1 

ISZ IY 


3447 

201 1 

ISZ IY 


3450 

2010 

ISZ IX 


3451 

2010 

ISZ IX 


3452 

2010 

ISZ IX 


3453 

2366 

ISZ NEGN 


3454 

5221 

JMP BEGIN 

/CONTINUE SUMMATION LOOP 

3455 

7200 

CL A 


3456 

1117 

TAD N 

/FLOAT N 

3457 

4777 

CALL FLOAT 


3460 

4407 

FLPT 


3461 

61 14 

FPUT G 


3462 

5175 

FGET SUMXX 

/CALC S AND B 

3463 

3114 

FMPY G 


3464 

6156 

FPUT TEM2 


346 5 

5164 

FGET SUMX 


3466 

3164 

FMPY SUMX 


3467 

2156 

FSUB TEM2 


3470 

6156 

FPUT TEM2 

/DENOMINATOR 

347 1 

5172 

FGET SUMXY 


3472 

31 14 

FMPY G 


3473 

6161 

FPUT TEM3 


3474 

5164 

FGET SUMX 


347 5 

3167 

FMPY SUMY 


3476 

2161 

FSUB TEM 3 


3477 

6111 

FPUT ERROR 

/NUMERATOR* TEMP STORAGE 

3500 

4156 

FDIV TEM2 

/SLOPE 

3501 

6100 

FPUT S 


3 502 

5175 

FGET SUMXX 


3503 

3167 

FMPY SUMY 


3 504 

6161 

FPUT TEM3 


3505 

5164 

FGET SUMX 


3506 

3172 

FMPY SUMXY 


3507 

2161 

FSUB TEM3 


3510 

4 156 

FDIV TEM2 


3511 

6103 

FPUT B 

/INTERCEPT 

3512 

5153 

FGET SUMYY 

/CALCULATE R 

3513 

3114 

FMPY G 


3514 

6161 

FPUT TEM3 


3515 

5167 

FGET SUMY 


3516 

3167 

FMPY SUMY 


3517 

2161 

FSUB TEM3 


3 520 

3156 

FMPY TEM2 


3 521 

0002 

FSQRT 



4 






3 522 

6161 


3523 

5111 


3524 

4161 


3525 

6106 


3 526 

0000 


3527 

7200 


3 530 

1 120 


3 531 

3010 


3 532 

1 121 


3533 

301 1 


3534 

1 122 


3535 

3012 


3 536 

1 123 


3537 

3013 


3 540 

1 1 17 


3541 

7041 


3542 

3776 


3543 

4407 


3544 

51 1 1 


3545 

6161 


3546 

5361 


3547 

6111 


3550 

6172 


3551 

6167 


3552 

0000 


3553 

4755 


3554 

5600 


3555 

3600 

CONT* 

3556 

0000 

ONE, 

3557 

3777 


3560 

7774 


3561 

0000 

ZERO, 

3562 

0000 


3563 

0000 


3564 

7777 

Ml, 

3565 

7776 

M2, 

3566 

0000 

NEGN, 

/PG 2 OF 

3576 

3666 

*3600 

3577 

4355 


3600 

0000 

LSQ2, 

3601 

4407 


3602 

5164 


3603 

4114 


3604 

6164 


3605 

0000 


3606 

4407 

LOOP, 

3607 

5410 


3610 

3100 


3611 

1 103 


3612 

6412 


3613 

2411 


3614 

6413 


3615 

34 13 


3616 

1 1 1 1 



FPUT I EM 3 
FGET ERROR 
FDIV TEM3 
FPUT R 
FEXT 
CL A 

TAD LX 
DCA IX 
TAD LY 
DCA IY 
TAD LCALC 
DCA I CALC 
TAD LDIF 
DCA I DIF 
TAD N 
CIA 

DCA NEGN2 
FLPT 

FGET ERROR 
FPUT TEM3 
FGET ZERO 
FPUT ERROR 
FPUT SUMXY 
FPUT SUMY 
FEXT 

CALL CONT 
RETURN LSQ 
3600 

FLTG 1.0 


FLTG 0.0 


- 1 
-2 
0 

LEAST SQUARES PROGRAM 


0 


FLPT 



FGET 

SUMX 

FDI V 

G 


FPUT 

SUMX 

FEXT 



FLPT 



FGET 

I 

IX 

FMPY 

S 


FADD 

B 


FPUT 

I 

I CALC 

F SUB 

I 

IY 

FPUT 

I 

IDIF 

FMPY 

I 

IDIF 

FADD 

ERROR 


/SET UP kEGISTERS 


/INDEX 


/NUMERATOR 


/ERROR SUMS 


/CALC RMS ERROR 
/AND ERROR IN SLOPE 
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3617 

6111 


FPUT ERROR 

/SUM C ERROR)**2 

3620 

5410 


FGET I IX 


3621 

2164 


F SUB SUMX 


3622 

3413 


FMPY I I DIF 


3623 

6156 


FPUT TEM2 


3624 

3156 


FMPY TEM2 


3625 

1 172 


FADD SUMXY 


3626 

6172 


FPUT SUMXY 

/SUM (X-XAVE)*< ERROR)**2 

3627 

0000 


FEXT 


3630 

2010 


ISZ IX 


3631 

2010 


ISZ IX 


3632 

2010 


ISZ IX 


3633 

201 1 


ISZ IY 


3634 

201 1 


ISZ IY 


3635 

2011 


ISZ IY 


3636 

2012 


ISZ I CALC 


3637 

2012 


ISZ I CALC 


3640 

2012 


ISZ I CALC 


3641 

2013 


ISZ I DIF 


3642 

2013 


ISZ I DIF 


3643 

2013 


ISZ IDIF 


3644 

2266 


ISZ NEGN2 


3645 

5206 


JMP LOOP 


3646 

4407 


FLPT 


3647 

511 1 


FGET ERROR 


36 50 

4114 


FDIV G 


3651 

0002 


FSQRT 


36 52 

6111 


FPUT ERROR 

/ERROR CALCULATED 

3653 

5172 


FGET SUMXY 


3654 

0002 


FSQRT 


3655 

31 14 


FMPY G 


3656 

4161 


FDIV TEM3 


3657 

3263 


FMPY HUNDRED 


3660 

6156 


FPUT TEM2 

/% ERROR IN SLOPE 

3661 

0000 


FEXT 


3662 

5600 


RETURN LSQ2 


3663 

0007 

HUNDRED, 

FLTG 100.0 


3664 

3077 




3665 

7776 




3666 

0000 

NEGN2, 

0 
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/SUBROUTINE TO FLOAT FIXED POINT NUMBERS 
/NUMBER IS IN AC; RETURNED IN FL(AC) 
*4355 


4355 

0000 

FLOAT, 

0 


4356 

3045 


DCA 45 

/PUT NUMBER IN HIGH ORDER 

4357 

3046 


DCA 46 

/PUT ZERO IN LOW ORDER 

4360 

1366 


TAD C13 

/11C10) INTO 

4361 

3044 

C j 

DCA 44 

/EXPONENT 

4362 

4407 


JMS I 7 

/CALL INTERPRETER 

4 36 3 

7000 


FNOR 

/NORMALIZE 

4364 

0000 


FEXT 

/LEAVE INTERPRETER 

4365 

5755 


JMP I FLOAT 


4366 

0013 

Cl 3, 

13 

/II DECIMAL 


(Cf. Floating-Point System Programming Manual p 3-10) 
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